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Introduction. The field of augmented reality (AR) is growing rapidly and has great advances in interaction, navigation 
and tracking. Nowadays there are a lot of trends for AR applications in different areas (education, entertainment, busi- 
ness, medicine, etc.). However, there is a lack of research to provide the evaluating AR apps framework to support de- 
velopers when creating suitable AR applications for specific needs. We provide a practical approach to quantify some 
of the AR applications features. We focus on the development of criteria for evaluating augmented reality applications. 
We discuss the criteria of choosing dimensions for that space such as standards for AR, tools for AR development, nav- 
igation and tracking, content management, usability. We provide analysis and evaluation of AR apps through each char- 
acteristic using guidelines which we have developed. 

Materials and Methods. An AR application is a software application that integrates digital visual, audio and other types 
of content into a real-world environment. The software quality and performance are the main characteristics of the ap- 
plication, which are key factors for AR applications. The analysis of scientific papers, documents and standards made it 
possible to determine characteristics that are the most significant quality indicators based on well-grounded users’ needs 
and demands. 

Results. The criteria we have developed for evaluating applications with augmented reality enable developers to create 
their own software products in stages, based on step-by-step requirements for them, evaluating the development process 
by characteristics. This approach will allow you to create high-quality software products using standardized, modern 
development tools. 

Discussion and Conclusions. In addition, developers will have a detailed understanding of each stage of creating the 
application and the necessary development tools and technologies to obtain the highest quality result. That will give an 
opportunity to decide on specific development tools, methods, models and technologies before starting work on a 
project. As a result, it will provide the final high-quality software product with good extensibility and compliance with 
the modern requirements of the digital industry market. 
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Introduction. The development of the modern market of augmented reality (AR) technology contributes to the 
emergence of a larger number of AR-applications widely used in industries including healthcare, public safety, gas and 
oil, tourism and marketing, entertainment and academia. In connection with the increased interest in this technology, the 
development of functional capabilities of AR projects is also carried out, which stimulates an increase in the needs of 
AR-technology using in the most diverse sectors of modern society. 

Unfortunately, some significant aspects of the AR applications development and implementation and AR ser- 
vices, often ignored for the design simplicity and implementing speed, are the compliance with such systems to real 
conditions and the evaluating under real operating conditions. To develop successful and highly efficient AR systems 
that can be adopted in everyday scenarios, user assessment and feedback are very important [1]. 

Augmented reality is a very young industry, and there are still no generally accepted standards for developing 
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AR applications. Although the main platforms for creating augmented reality applications have been defined: ARKit', 
ARCore’ and Unity*® (mostly used) — even now AR applications developed using these tools are available for launch 
only on a limited type of digital and mobile devices. 

The situation is exactly the same with wearable devices, HoloLens and Magic Leap glasses — each manufactur- 
er offers its own unique software for creating augmented reality programs. Experts give several years before the indus- 
try consolidates and common standards allow making the development of augmented reality applications accessible and 
universal for all AR developers. 

Rapid progression of the AR field requires effective and validated methods of design evaluation to be devel- 
oped. Failure to consider the usability of AR applications during the design process will result in an increase in user 
errors and accidents, limiting user trust of the technology and undermining user perceptions of the technology, for both 
AR and Virtual Reality (VR) technologies [2]. 

The authors of the papers [1—3] provided an overview of the important designing and implementing features of 
AR applications and proposed theoretical evaluation of AR systems and frameworks through the standardization as- 
pects. Endsley and others [4] described principles of design heuristics for AR for multi-dimensional augmented envi- 
ronments. Some examples of user experience evaluations were presented in the papers [5—10]. 

J. L. Gabbard and J. E. Swan [11] proposed a Usability Engineering (UE) for Augmented Reality approach that 
inserts iteratively a series of user-based studies into a traditional usability-engineering life cycle. Several usability test- 
ing methods of the AR application (subjective measurement using human perception, objective measure from observa- 
tion, evaluation by expert through cognitive walkthrough, heuristic evaluation, lab observation, questionnaire) were 
described by Pranoto and others [12]. Martins and others [13] presented practical use of the usability methods for evalu- 
ating an AR children’s book with multiple methods. Other aspects of AR technology tools and AR applications evaluat- 
ing were presented in the papers [14] and [15]. 

Akgul and others [16] adapted an existing deep learning architecture to solve the detection problem in AR ap- 
plication using camera-based tracking. Other methods that help to improve AR application and to increment the produc- 
tivity in manufacture were described in [17-19]. 

There are several survey papers on AR development, but none is dedicated to Mobile Augmented Reality. 
Huang and others [20] present the results of the latest technologies and methods survey that improves run-time perfor- 
mance and energy efficiency for the practical implementation of mobile AR applications. 

We have presented AR application “Tilsimli arifler” (“Magic letters”) and special features of design and de- 
veloping mobile AR application for enhancing early literacy skills in our papers [21—22]. 

Software quality and performance. An AR application is a software application that integrates digital visual, 
audio and other types of content into a real-world environment. The software quality and performance are the main 
characteristics of the application, which are key factors for AR applications. Software development is a complex and 
multi-faceted process, in which a large number of specialists of various areas of expertise and various skill levels partic- 
ipate. In addition, many technical, technological, and managerial issues intertwine in the application development pro- 
cess. The success of the project and the quality of the developed product depend on their adequate involvement. 

The analysis of papers, documents (e.g., [2, 3, 10, 16, 22]) and standards (e.g., ISO-IEC JTC 1 SC 24, 18521- 
1*, ISO 9000°) shows that the following characteristics are the key quality indicators based on well-grounded users’ 
needs and demands: 

— inadequate functioning of the software product; 

— insufficient interaction of the product with other software, hardware, telecommunications; 

— failures of the software application during the intended use; 

— the slowed down time of the software product and the delay in the presentation of intermediate and output 
information; 

— incomplete display of information; 

— inconsistency of stored data and information entered by the operator; 

— loss of relevance of the information; 

— violation of the confidentiality of information. 

In addition to such “primary” quality data coming directly from the consumer, developers use “internal” indi- 
cators to evaluate the parameters of the current project: 


' Augmented Reality — Apple Developer. https://developer.apple.com/augmented-reality/ 

* ARCore — Google Developers. https://developers.google.com/ar 

> Unity for all. https://unity.com/ 

4 Kim G. Augmented Reality Continuum Concepts and Reference Model — Part 1: ARC Reference Model (Work Item Proposal), ISO-IEC JTC 1 SC 
24, 18521-1. 2012. 

> ISO 9000 Family Quality Management. https://www.iso.org/iso-900 1 -quality-management.html 
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— the lines of code in the standard mode; 

— the number of detected errors per 1000 lines of code; 

— program complexity parameters; 

— the probability of occurrence of specific errors; 

— the project complexity and cost of a code unit; 

— price of a “man-month”; 

— statistical characteristics of processes (expectation, variance, correlation function, etc.) and other estimated 
parameters. 

Augmented Reality Application Evaluation Criteria. Ten experts with more than 3 years’ experience in AR 
application development help us to create criteria for evaluating AR apps. We divided the main criteria for assessing the 
quality of augmented reality applications into 5 groups (Fig 1): 

— AR app design and Art design; 

— Graphic programming; 

— AR applications programming; 

— Application profiling and optimization; 

— Publishing applications (build). 

When developing an AR application design, we recommend the following characteristics that should be con- 
sidered: 

1. The surface. How the application can adapt to various surfaces. If you use frameworks, you can use the 
built-in surface detection function. For example, the application can recognize the floor, walls and objects. 

2. Shine. Evaluation of illumination is very important for the realism of objects. It is advisable to use dynamic 
lighting with shadows in real time. 

3. Space needed. Users can experience AR in three different sizes: table scale, room scale, and open environ- 
ment. The user should always have enough space to simply enjoy the experience. Thus, it is necessary to think over this 
before the user starts using the application. For example, if an application requires an open environment, you need to 
inform the audience in advance before they start using the product. 

4. Single-user or multi-user product. If you are developing a multi-user interface, you should design the prod- 
uct when everyone is involved in the process. It is important to create a sense of audience connection with the product. 
For example, if you are developing an AR game, you can provide a map that shows the location of users and provides 
real-time status notifications. 











Fig. 1. Evaluation criteria for AR applications 


The main criteria are highlighted as follows (Table 1). 
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Table | 
AR App Design and Art Design evaluation points 





AR App Design and Art Design 
Actual requirements (of the market) in terms of application design 


N 
Nn 








Interface design requirements 
UX features for AR applications 








Target platform specifications 
3D modeling principles 





Principles of work with particle systems 
Types and features of the creation and use of textures 





NP Ry RlRelmyRelNMpsls 


Principles of working with shaders and materials, rendering features 





Features for customizing models and textures, as well as materials for export to the game 
engine 
Create high-quality 3D models in modern editors 
Create UV-scan in modern editors 


ee 








Texturize models in modern editors 





Rigging and animating models in modern editors 





Customize animation in game engines 





NPE NMPEN] NM] dO] dy 


Create and customize shaders, materials in simulation programs and game engines 














The initial version of Table 1 was proposed by the authors, then it was discussed with ten experts in AR devel- 
opment using brainstorming techniques. 

To solve the problems of displaying complex graphic objects, it is necessary to develop effective methods and 
algorithms for processing graphic information at the stages of input, encoding, transformation and image formation. All 
of this makes up a set of basic computer graphics tasks, so we have identified the main components needed to create a 
high-quality graphic component (Table 2). 

Graphic programming enables to create visual effects through the development of shaders using the functions 
of graphic libraries, customizing the rendering of the development environment, using effective postprocessing librar- 
ies, customizing shadows, and more. All these components make it possible to obtain high-quality images by applying 
various effects, as well as improve the optimization of application performance. 

Table 2 
Graphic programming evaluation points 





N 
Nn 


Graphic programming 





Rendering features on the target platform 





Using textures and materials in the game engine 
Features of the work of graphic libraries 
Principles of rendering geometric objects and images 
Principles of proper postprocessing 





Features of implementing lighting and shadows on the used game engine and target platform 





Configure static and dynamic lighting in the used game engine 





Optimize rendering processes 





Customize postprocessing and final image appearance 





Create procedural geometry using game engine tools 














NP NEN] WIM] W] WY] ROT NT] dO] 


Programming the rendering of the frame 





Evaluation points presented in Table 2 were proposed by the authors and later they were discussed with ten 
experts in AR development. 

During the development of the software part of the mobile application with augmented reality, it is necessary 
to precisely determine the fundamental development tools, and, accordingly, the programming languages, programming 
environments, patterns, architecture, etc. This is necessary for a clearer and more coordinated team work, understanding 


the requirements and tasks, comprehension of the logic and sequence of application development, orientation in the 
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project, reducing risks, and interaction of each team member with each other. To achieve this, we have identified the 
following points for the programming phase, which should be followed in order to obtain a positive result of the devel- 
opment process (Table 3). 


Table 3 
AR Applications Programming evaluation points 





N 
Nn 


AR Applications Programming 
Modern programming languages used in AR development 
OOP principles 
Building an application architecture 














Principles of building AR applications 
Code design standards 
Basic math for AR applications 
Work with a network in the context of AR 
Work with modern AR application development environments 
Write high-quality code in modern programming languages used in AR 














Implement specific application mechanics as soon as possible 
Use development environments for writing and debugging code 
Work with necessary SDKs for AR 
Use collaboration tools 














RFIPNIENMI NMI NMIMPNM! MN] NM] NM] NM] NH] tv 








The evaluation points presented in Table 3 were proposed by the authors and brainstormed with ten experts in 
AR development. 

When developing applications with augmented reality, it is important to consider the features of application 
optimization for PC and mobile devices, as well as the architecture of mobile devices in the context of application opti- 
mization. In this regard, we have identified the points that we advise to adhere to during the development in order to 
achieve maximum optimization of AR applications (Table 4). 

The profiler tool provides specific data on game performance and facilitates its optimization process. The pro- 


filer provides frame-by-frame metrics with which you can more easily identify problem areas. 
Table 4 
Application Profiling and Optimization evaluation points 





Application Profiling and Optimization 16 
Optimization of AR application performance 
Optimization of 3D objects 











Geometry optimization 
Optimization of textures and materials for the target platforms 








Optimization of the main application processes 
Optimization of physics in the application 
Using built-in game engine profilers, as well as external profilers 
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Table 4 was created by the authors and then it was discussed with ten experts in AR development. 

After completion of all stages of the application development, it is important to publish the application. The 
publication represents the release of the application on any platform where the customer (end-user) can easily download 
the final version of product, get acquainted with it, get all the necessary documentations, technical support and feedback 
from the developer. Each of the platforms puts forward its specific requirements to the publishing application, which are 
necessary for correct displaying the application in the platform’s market, obtaining all information about the applica- 
tion’s operation, ensuring end-user security, promoting the application, and more. In this regard, the publication is one 
of the most important and crucial stage of the development. We have identified several main platforms, before using 
which it is necessary to familiarize yourself with all the documentation and assembly features for the appropriate plat- 
form (Table 5). 
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Table 5 
Publishing applications (build) evaluation points 





Publishing applications (build) 





Features of building an application for Windows 





Features of the build application for Android. 
Features of the build application for OS X/ iOS. 





WW] Wy] Wy] oO 














Evaluation points presented in Table 5 were proposed by the experts in AR development. 

Conclusions. Nowadays, the augmented reality is one of the most innovative and a new digital trend in the de- 
veloping applications for all type of devices. The AR technology opens a new horizon and is going to get more popular 
in the foreseeable future. 

The criteria we have developed for evaluating applications with augmented reality enable developers to create 
their own software products in stages, based on step-by-step requirements for them. This will allow you to create high- 
quality software products using the standardized, modern development tools. 

In addition, developers will have a detailed understanding of each stage of creating the application and the 
necessary development tools and technologies to obtain the highest quality result. That will give an opportunity to de- 
cide on specific development tools, methods, models and technologies before starting work on a project; and, in the 
process of working on the basis of existing criteria, gradually create key application stages, with possible subsequent 
upgrades and improvements. As a result, it will provide the final high-quality software product with good extensibility 
and compliance with the modern requirements of the digital industry market. 

In the follow-up study, we are going to apply this approach for evaluating several AR applications. To thor- 
oughly verify the proposed criteria, additional testing will be required, where more software field experts should be 
involved. Engaging third-party experts will assess the suitability of the proposed criteria. We believe that this approach 


will become part of the AR application development process. 
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